{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 基于千帆与 FLAML 的模型配置自动推荐\n",
    "\n",
    "千帆 SDK 提供了模型和模型配置自动推荐的功能，用户只需要提供目标场景的数据集和评估器，指定搜索空间，SDK 就可以自动推荐出最佳的模型配置。\n",
    "\n",
    "FLAML 是一个开源的 AutoML 库，提供了许多优秀的参数推荐算法，可以高效地去寻找最佳的模型配置。\n",
    "\n",
    "然而，由于 FLAML 本身并不支持千帆 SDK 的数据集和评估器，这需要额外的开发工作。对此，千帆 SDK 为 FLAML 提供了适配层，可以快速基于千帆和 FLAML 的算法实现模型配置自动推荐。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install qianfan\n",
    "!pip install flaml[blendsearch]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import qianfan\n",
    "\n",
    "os.environ[\"QIANFAN_ACCESS_KEY\"] = \"your_access_key\"\n",
    "os.environ[\"QIANFAN_SECRET_KEY\"] = \"your_secret_key\"\n",
    "# 由于后续在调优配置的过程中会并发请求模型，建议限制 QPS 和重试次数，避免调用失败\n",
    "os.environ[\"QIANFAN_QPS_LIMIT\"] = \"3\"\n",
    "os.environ[\"QIANFAN_LLM_API_RETRY_COUNT\"] = \"5\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备工作\n",
    "\n",
    "与直接使用千帆 SDK 的参数推荐功能相同，我们需要先准备：\n",
    "\n",
    "- 数据集 Dataset：根据目标场景准备一定量的数据\n",
    "- 评估方式 Evaluator：根据目标场景，选择待优化的指标，并提供评估函数\n",
    "\n",
    "数据集使用的是千帆 SDK 中提供的 Dataset 模块，可以直接加载本地的数据集文件，也可以使用平台上预置的或者自行上传的数据集，具体加载方式参考 [文档](https://github.com/baidubce/bce-qianfan-sdk/blob/main/docs/dataset.md)。\n",
    "\n",
    "评估采用的 SDK 提供的 Evaluator 模块，基于 Evaluator 实现 evaluate 方法即可，关于如何实现 Evaluator 可以参考 [该cookbook](https://github.com/baidubce/bce-qianfan-sdk/blob/main/cookbook/evaluation/local_eval_with_qianfan.ipynb)。\n",
    "\n",
    "如下加载了本地的一个数据集，并实现了一个利用大模型评分实现评估的 Evaluator，如果你对基础的使用方式还不熟悉，可以参考 [该cookbook](https://github.com/baidubce/bce-qianfan-sdk/blob/main/cookbook/autotuner/tune.ipynb)，此处省略相关的介绍。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [04-25 15:25:52] dataset.py:408 [t:139839157186816]: no data source was provided, construct\n",
      "[INFO] [04-25 15:25:52] dataset.py:276 [t:139839157186816]: construct a file data source from path: ./example.jsonl, with args: {'input_columns': ['prompt'], 'reference_column': 'response'}\n",
      "[INFO] [04-25 15:25:52] file.py:298 [t:139839157186816]: use format type FormatType.Jsonl\n",
      "[INFO] [04-25 15:25:52] utils.py:416 [t:139839157186816]: need create cached arrow file for /root/work/bce-qianfan-sdk/cookbook/autotuner/example.jsonl\n",
      "[INFO] [04-25 15:25:52] utils.py:461 [t:139839157186816]: start to write arrow table to /root/.qianfan_cache/dataset/root/work/bce-qianfan-sdk/cookbook/autotuner/example.arrow\n",
      "[INFO] [04-25 15:25:52] utils.py:473 [t:139839157186816]: writing succeeded\n",
      "[INFO] [04-25 15:25:52] utils.py:347 [t:139839157186816]: start to get memory_map from /root/.qianfan_cache/dataset/root/work/bce-qianfan-sdk/cookbook/autotuner/example.arrow\n",
      "[INFO] [04-25 15:25:52] utils.py:275 [t:139839157186816]: has got a memory-mapped table\n"
     ]
    }
   ],
   "source": [
    "from qianfan.dataset import Dataset\n",
    "from qianfan.evaluation.evaluator import LocalEvaluator\n",
    "from qianfan import ChatCompletion\n",
    "from qianfan.common.prompt.prompt import Prompt\n",
    "from qianfan.utils.pydantic import Field\n",
    "\n",
    "from typing import Optional, Union, Any, Dict, List\n",
    "import re\n",
    "import json\n",
    "\n",
    "dataset = Dataset.load(\n",
    "    data_file=\"./example.jsonl\",\n",
    "    organize_data_as_group=False,\n",
    "    input_columns=[\"prompt\"],\n",
    "    reference_column=\"response\",\n",
    ")\n",
    "\n",
    "class LocalJudgeEvaluator(LocalEvaluator):\n",
    "    model: Optional[ChatCompletion] = Field(default=None, description=\"model object\")\n",
    "    cache: Dict[str, Any] = Field(default={}, description=\"cache for evaluation\")\n",
    "    eval_prompt: Prompt = Field(\n",
    "        default=Prompt(\n",
    "            template=\"\"\"你需要扮演一个裁判的角色，对一段角色扮演的对话内容进行打分，你需要考虑这段文本中的角色沉浸度和对话文本的通畅程度。你可以根据以下规则来进行打分，你可以阐述你对打分标准的理解后再给出分数：\n",
    "                \"4\":完全可以扮演提问中的角色进行对话，回答完全符合角色口吻和身份，文本流畅语句通顺\n",
    "                \"3\":扮演了提问中正确的角色，回答完全符合角色口吻和身份，但文本不流畅或字数不满足要求\n",
    "                \"2\":扮演了提问中正确的角色，但是部分语句不符合角色口吻和身份，文本流畅语句通顺\n",
    "                \"1\":能够以角色的口吻和身份进行一部分对话，和角色设定有一定偏差，回答内容不流畅，或不满足文本字数要求\n",
    "                \"0\":扮演了错误的角色，没有扮演正确的角色，角色设定和提问设定差异极大，完全不满意\n",
    "                你的回答需要以json代码格式输出：\n",
    "                ```json\n",
    "                {\"modelA\": {\"justification\": \"此处阐述对打分标准的理解\", \"score\": \"此处填写打分结果\"}}\n",
    "                ```\n",
    "\n",
    "                现在你可以开始回答了：\n",
    "                问题：{{input}}\n",
    "                ---\n",
    "                modelA回答：{{output}}\n",
    "                ---\"\"\",\n",
    "            identifier=\"{{}}\",\n",
    "        ),\n",
    "        description=\"evaluation prompt\",\n",
    "    )\n",
    "\n",
    "    class Config:\n",
    "        arbitrary_types_allowed = True\n",
    "\n",
    "    def evaluate(\n",
    "        self, input: Union[str, List[Dict[str, Any]]], reference: str, output: str\n",
    "    ) -> Dict[str, Any]:\n",
    "        score = 0\n",
    "        try:\n",
    "            # 渲染评估用的 prompt，传入输入、模型输出和参考答案\n",
    "            p, _ = self.eval_prompt.render(\n",
    "                **{\n",
    "                    \"input\": \"\\n\".join([i[\"content\"] for i in input[1:]]),\n",
    "                    \"output\": output,\n",
    "                    \"expect\": reference,\n",
    "                }\n",
    "            )\n",
    "            # 利用 cache 避免对同一结果反复进行评估，提升效率\n",
    "            if p in self.cache:\n",
    "                model_output = self.cache[p]\n",
    "                score = float(model_output[\"modelA\"][\"score\"])\n",
    "            else:\n",
    "                # 请求模型进行评估\n",
    "                r = self.model.do(messages=[{\"role\": \"user\", \"content\": p}], temperature=0.01)\n",
    "                content = r[\"result\"]\n",
    "                model_output = content\n",
    "                # 提取出 json 格式的评估结果\n",
    "                regex = re.compile(\"\\`\\`\\`json(.*)\\`\\`\\`\", re.MULTILINE | re.DOTALL)\n",
    "    \n",
    "                u = regex.findall(content)\n",
    "    \n",
    "                if len(u) == 0:\n",
    "                    score = 0\n",
    "                else:\n",
    "                    model_output = json.loads(u[0])\n",
    "                    score = float(model_output[\"modelA\"][\"score\"])\n",
    "                    self.cache[p] = model_output\n",
    "        except Exception as e:\n",
    "            score = 0\n",
    "        # 返回评估结果，这里字段需与后续推荐配置时设定的评估字段一致\n",
    "        return {\"score\": score}\n",
    "\n",
    "    def summarize(self, metric_dataset: Dataset) -> Optional[Dict[str, Any]]:\n",
    "        score_sum = 0\n",
    "        count = 0\n",
    "\n",
    "        for line in metric_dataset.list():\n",
    "            score_sum += line[\"score\"]\n",
    "            count += 1\n",
    "\n",
    "        return {\"score\": score_sum / float(count)}\n",
    "\n",
    "local_evaluator = LocalJudgeEvaluator(\n",
    "    model=ChatCompletion(model=\"ERNIE-Bot-4\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在开始调优前，我们还需要选择调优的算法和相关配置，这部分使用的是 FLAML 提供的 Searcher，我们需要准备一个 searcher，如下以 BlendSearch 为例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You passed a `space` parameter to OptunaSearch that contained unresolved search space definitions. OptunaSearch should however be instantiated with fully configured search spaces only. To use Ray Tune's automatic search space conversion, pass the space definition as part of the `config` argument to `tune.run()` instead.\n",
      "\u001b[32m[I 2024-04-25 15:42:06,661]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from flaml import BlendSearch, tune\n",
    "\n",
    "blendsearch = BlendSearch(\n",
    "    metric=\"score\",  # 指标，与 Evalutor 返回结果对应\n",
    "    mode=\"max\",\n",
    "    space={          # 搜索空间\n",
    "        \"temperature\": tune.uniform(0.01, 0.99),\n",
    "        \"model\": tune.choice(\n",
    "            [\n",
    "                \"ERNIE-Speed\",\n",
    "                \"ERNIE-Bot-turbo\",\n",
    "            ]\n",
    "        ),\n",
    "    },\n",
    "    time_budget_s=600,       # 时间约束\n",
    "    cost_attr=\"total_cost\",  # 设置成本的 key，使用千帆 SDK 时该 key 为 total_cost\n",
    "    cost_budget=20,          # 成本约束，单位为 元\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 开始调优\n",
    "\n",
    "之后我们就可以传入参数开始寻找最有的模型配置了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [04-25 15:44:48] launcher.py:108 [t:139839157186816]: turn 0 started...\n",
      "[INFO] [04-25 15:44:48] launcher.py:109 [t:139839157186816]: suggested config list: [{'temperature': 0.765894230401411, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-25 15:44:48] dataset.py:994 [t:139839157186816]: list local dataset data by None\n",
      "[INFO] [04-25 15:48:09] launcher.py:114 [t:139839157186816]: config: {'temperature': 0.765894230401411, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.21, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 106.42, 'avg_total_tokens': 861.22, 'avg_req_latency': 2.439878113121231, 'avg_tokens_per_second': 352.97664886147874, 'avg_cost': 0.0029029200000000007, 'total_cost': 0.29029200000000005, 'success_rate': 1.0, 'total_time': 201.03477883338928}\n",
      "[INFO] [04-25 15:48:09] launcher.py:108 [t:139839157186816]: turn 1 started...\n",
      "[INFO] [04-25 15:48:09] launcher.py:109 [t:139839157186816]: suggested config list: [{'temperature': 0.7438278048878396, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-25 15:48:09] dataset.py:994 [t:139839157186816]: list local dataset data by None\n",
      "[INFO] [04-25 15:51:06] launcher.py:114 [t:139839157186816]: config: {'temperature': 0.7438278048878396, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.67, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 55.75, 'avg_total_tokens': 749.25, 'avg_req_latency': 1.8456731940702593, 'avg_tokens_per_second': 405.9494402406531, 'avg_cost': 0.0032200000000000006, 'total_cost': 0.32200000000000006, 'success_rate': 1.0, 'total_time': 177.0761489868164}\n",
      "[INFO] [04-25 15:51:06] launcher.py:108 [t:139839157186816]: turn 2 started...\n",
      "[INFO] [04-25 15:51:06] launcher.py:109 [t:139839157186816]: suggested config list: [{'temperature': 0.2041016074644315, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-25 15:51:06] dataset.py:994 [t:139839157186816]: list local dataset data by None\n",
      "[INFO] [04-25 15:53:37] launcher.py:114 [t:139839157186816]: config: {'temperature': 0.2041016074644315, 'model': 'ERNIE-Speed'}, metrics: {'score': 3.21, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 44.59, 'avg_total_tokens': 738.09, 'avg_req_latency': 1.589856743550772, 'avg_tokens_per_second': 464.249375293749, 'avg_cost': 0.0031307199999999996, 'total_cost': 0.31307199999999996, 'success_rate': 1.0, 'total_time': 150.52750873565674}\n",
      "[INFO] [04-25 15:53:37] launcher.py:108 [t:139839157186816]: turn 3 started...\n",
      "[INFO] [04-25 15:53:37] launcher.py:109 [t:139839157186816]: suggested config list: [{'temperature': 0.09657301789053005, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-25 15:53:37] dataset.py:994 [t:139839157186816]: list local dataset data by None\n",
      "[INFO] [04-25 15:56:33] launcher.py:114 [t:139839157186816]: config: {'temperature': 0.09657301789053005, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.56, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 125.19, 'avg_total_tokens': 879.99, 'avg_req_latency': 2.6666990189495845, 'avg_tokens_per_second': 329.9922464990552, 'avg_cost': 0.003015540000000001, 'total_cost': 0.3015540000000001, 'success_rate': 1.0, 'total_time': 176.00841450691223}\n",
      "[INFO] [04-25 15:56:33] launcher.py:108 [t:139839157186816]: turn 4 started...\n",
      "[INFO] [04-25 15:56:33] launcher.py:109 [t:139839157186816]: suggested config list: [{'temperature': 0.3425908632261716, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-25 15:56:33] dataset.py:994 [t:139839157186816]: list local dataset data by None\n",
      "[INFO] [04-25 15:58:53] launcher.py:114 [t:139839157186816]: config: {'temperature': 0.3425908632261716, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.85, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 44.41, 'avg_total_tokens': 737.91, 'avg_req_latency': 1.7740423735498916, 'avg_tokens_per_second': 415.9483510664001, 'avg_cost': 0.0031292800000000008, 'total_cost': 0.3129280000000001, 'success_rate': 1.0, 'total_time': 140.53086590766907}\n",
      "[INFO] [04-25 15:58:53] launcher.py:108 [t:139839157186816]: turn 5 started...\n",
      "[INFO] [04-25 15:58:53] launcher.py:109 [t:139839157186816]: suggested config list: [{'temperature': 0.013869301001356162, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-25 15:58:53] dataset.py:994 [t:139839157186816]: list local dataset data by None\n",
      "[INFO] [04-25 16:01:35] launcher.py:114 [t:139839157186816]: config: {'temperature': 0.013869301001356162, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.385, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 109.89, 'avg_total_tokens': 864.69, 'avg_req_latency': 2.4941846013496978, 'avg_tokens_per_second': 346.6824386342869, 'avg_cost': 0.0029237399999999998, 'total_cost': 0.29237399999999997, 'success_rate': 1.0, 'total_time': 161.29327535629272}\n",
      "[INFO] [04-25 16:01:35] launcher.py:108 [t:139839157186816]: turn 6 started...\n",
      "[INFO] [04-25 16:01:35] launcher.py:109 [t:139839157186816]: suggested config list: [{'temperature': 0.6102755454928004, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-25 16:01:35] dataset.py:994 [t:139839157186816]: list local dataset data by None\n",
      "[INFO] [04-25 16:04:09] launcher.py:114 [t:139839157186816]: config: {'temperature': 0.6102755454928004, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.75, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 44.9, 'avg_total_tokens': 738.4, 'avg_req_latency': 1.6426457330596167, 'avg_tokens_per_second': 449.5187155325604, 'avg_cost': 0.0031332, 'total_cost': 0.31332, 'success_rate': 1.0, 'total_time': 154.09031200408936}\n",
      "[INFO] [04-25 16:04:09] launcher.py:108 [t:139839157186816]: turn 7 started...\n",
      "[INFO] [04-25 16:04:09] launcher.py:109 [t:139839157186816]: suggested config list: [{'temperature': 0.9094186400626846, 'model': 'ERNIE-Speed'}]\n",
      "[INFO] [04-25 16:04:09] dataset.py:994 [t:139839157186816]: list local dataset data by None\n",
      "[INFO] [04-25 16:07:04] launcher.py:114 [t:139839157186816]: config: {'temperature': 0.9094186400626846, 'model': 'ERNIE-Speed'}, metrics: {'score': 2.44, 'avg_prompt_tokens': 693.5, 'avg_completion_tokens': 61.84, 'avg_total_tokens': 755.34, 'avg_req_latency': 2.005694531320478, 'avg_tokens_per_second': 376.59772622639156, 'avg_cost': 0.0032687200000000006, 'total_cost': 0.32687200000000005, 'success_rate': 1.0, 'total_time': 175.2104778289795}\n",
      "[INFO] [04-25 16:07:04] launcher.py:108 [t:139839157186816]: turn 8 started...\n",
      "[INFO] [04-25 16:07:04] launcher.py:109 [t:139839157186816]: suggested config list: [{'temperature': 0.14932664664949644, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-25 16:07:04] dataset.py:994 [t:139839157186816]: list local dataset data by None\n",
      "[INFO] [04-25 16:09:44] launcher.py:114 [t:139839157186816]: config: {'temperature': 0.14932664664949644, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.475, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 118.96, 'avg_total_tokens': 873.76, 'avg_req_latency': 2.5422776999295458, 'avg_tokens_per_second': 343.691800476484, 'avg_cost': 0.0029781599999999997, 'total_cost': 0.29781599999999997, 'success_rate': 1.0, 'total_time': 159.95888233184814}\n",
      "[INFO] [04-25 16:09:44] launcher.py:108 [t:139839157186816]: turn 9 started...\n",
      "[INFO] [04-25 16:09:44] launcher.py:109 [t:139839157186816]: suggested config list: [{'temperature': 0.06561235170269139, 'model': 'ERNIE-Bot-turbo'}]\n",
      "[INFO] [04-25 16:09:44] dataset.py:994 [t:139839157186816]: list local dataset data by None\n",
      "[INFO] [04-25 16:12:12] launcher.py:114 [t:139839157186816]: config: {'temperature': 0.06561235170269139, 'model': 'ERNIE-Bot-turbo'}, metrics: {'score': 2.31, 'avg_prompt_tokens': 754.8, 'avg_completion_tokens': 121.43, 'avg_total_tokens': 876.23, 'avg_req_latency': 2.550379991089576, 'avg_tokens_per_second': 343.56841061384586, 'avg_cost': 0.002992979999999999, 'total_cost': 0.2992979999999999, 'success_rate': 1.0, 'total_time': 148.54167294502258}\n",
      "[INFO] [04-25 16:12:12] launcher.py:92 [t:139839157186816]: max turn reached: 10\n",
      "[INFO] [04-25 16:12:12] launcher.py:102 [t:139839157186816]: tuning finished!\n",
      "[INFO] [04-25 16:12:12] launcher.py:104 [t:139839157186816]: best config: {'temperature': 0.2041016074644315, 'model': 'ERNIE-Speed'}\n"
     ]
    }
   ],
   "source": [
    "from qianfan.extensions.flaml.autotuner import run\n",
    "\n",
    "context = await run(\n",
    "    searcher=blendsearch,\n",
    "    dataset=dataset,\n",
    "    evaluator=local_evaluator,\n",
    "    # 以下均为可选参数\n",
    "    repeat=5,            # 重复推理次数，用于减少大模型输出随机性对结果准确性的干扰\n",
    "    max_turn=10,         # 设定最大尝试次数\n",
    "    # max_time=3600,     # 设定最大尝试时间，单位为秒\n",
    "    log_dir= \"./log\",    # 日志目录\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以通过如下方式获取最佳的参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'temperature': 0.2041016074644315, 'model': 'ERNIE-Speed'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "context.best"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "或者通过如下方式获取每次尝试的配置及结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERNIE-Bot-turbo\t0.765894230401411\t2.21\n",
      "ERNIE-Speed\t0.7438278048878396\t2.67\n",
      "ERNIE-Speed\t0.2041016074644315\t3.21\n",
      "ERNIE-Bot-turbo\t0.09657301789053005\t2.56\n",
      "ERNIE-Speed\t0.3425908632261716\t2.85\n",
      "ERNIE-Bot-turbo\t0.013869301001356162\t2.385\n",
      "ERNIE-Speed\t0.6102755454928004\t2.75\n",
      "ERNIE-Speed\t0.9094186400626846\t2.44\n",
      "ERNIE-Bot-turbo\t0.14932664664949644\t2.475\n",
      "ERNIE-Bot-turbo\t0.06561235170269139\t2.31\n"
     ]
    }
   ],
   "source": [
    "for turn in context.history:\n",
    "    for trial in turn:\n",
    "        metrics = trial.metrics\n",
    "        config = trial.config\n",
    "        print(\"{}\\t{}\\t{}\".format(config['model'], config['temperature'], metrics['score']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此我们就完成了基于千帆 SDK 和 FLAML 的模型配置自动推荐。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py311",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
